Real-time and Freehand Multimodal Imaging: Combining White Light Endoscopy with All-Optical Ultrasound

Minimally invasive surgery offers significant benefits over open surgery in terms of patient recovery, complication rates, and cost. Accurate visualisation is key for successful interventions; however, no single imaging modality offers sufficient resolution, penetration, and soft-tissue contrast to adequately monitor interventional treatment. Consequently, multimodal interventional imaging is intensively investigated. All-optical ultrasound (AOUS) imaging is an emerging modality where light is used to both generate and detect ultrasound. Using fibre-optics, highly miniaturised imaging probes can be fabricated that yield high-quality pulse-echo images and are readily integrated into minimally invasive interventional instruments. In this work, we present the integration of a miniature (diameter: 800 µm), highly directional AOUS imaging probe into a commercially available white light urethroscope, and demonstrate the first real-time, 3D multimodal imaging combining AOUS and white light endoscopy. Through the addition of an electromagnetic tracker, the position and pose of the instrument could be continuously recorded. This facilitated accurate real-time registration of the imaging modalities, as well as freehand operation of the instrument. In addition, the freehand imaging paradigm allowed for “piece-wise” scanning where the instrument was retracted and repositioned without recalibration. The presented imaging probe and system could significantly improve the quality of image guidance during interventional surgery

Photoacoustic imaging of intracardiac medical devices using internal illumination of carbon nanotube / PDMS composite coatings

Accurate localisation of medical devices is of crucial importance for a wide range of ultrasound-guided interventions. In this study, we investigated visualisation of medical devices by photoacoustic excitation of optically absorbing coatings. Photoacoustic excitation light was provided through optical fibres positioned within a cardiac needle and a steerable-tip catheter. Using a swine heart model, photoacoustic and B-mode ultrasound images were received with a clinical ultrasound scanner in conjunction with a transoesophageal imaging probe. In the photoacoustic images, prominent signals were obtained from the coatings. This study demonstrated that photoacoustic imaging could play a useful role with medical device imaging

Patient-specific polyvinyl alcohol phantom fabrication with ultrasound and x-ray contrast for brain tumor surgery planning

Phantoms are essential tools for clinical training, surgical planning and the development of novel medical devices. However, it is challenging to create anatomically accurate head phantoms with realistic brain imaging properties because standard fabrication methods are not optimized to replicate any patient-specific anatomical detail and 3D printing materials are not optimized for imaging properties. In order to test and validate a novel navigation system for use during brain tumor surgery, an anatomically accurate phantom with realistic imaging and mechanical properties was required. Therefore, a phantom was developed using real patient data as input and 3D printing of molds to fabricate a patient-specific head phantom comprising the skull, brain and tumor with both ultrasound and X-ray contrast. The phantom also had mechanical properties that allowed the phantom tissue to be manipulated in a similar manner to how human brain tissue is handled during surgery. The phantom was successfully tested during a surgical simulation in a virtual operating room. The phantom fabrication method uses commercially available materials and is easy to reproduce. The 3D printing files can be readily shared, and the technique can be adapted to encompass many different types of tumor

Deep Learning for Instrumented Ultrasonic Tracking: From synthetic training data to in vivo application

Instrumented ultrasonic tracking is used to improve needle localisation during ultrasound guidance of minimally-invasive percutaneous procedures. Here, it is implemented with transmitted ultrasound pulses from a clinical ultrasound imaging probe that are detected by a fibre-optic hydrophone integrated into a needle. The detected transmissions are then reconstructed to form the tracking image. Two challenges are considered with the current implementation of ultrasonic tracking. First, tracking transmissions are interleaved with the acquisition of B-mode images and thus, the effective B-mode frame rate is reduced. Second, it is challenging to achieve an accurate localisation of the needle tip when the signal-to-noise ratio is low. To address these challenges, we present a framework based on a convolutional neural network (CNN) to maintain spatial resolution with fewer tracking transmissions and to enhance signal quality. A major component of the framework included the generation of realistic synthetic training data. The trained network was applied to unseen synthetic data and experimental in vivo tracking data. The performance of needle localisation was investigated when reconstruction was performed with fewer (up to eight-fold) tracking transmissions. CNN-based processing of conventional reconstructions showed that the axial and lateral spatial resolution could be improved even with an eight-fold reduction in tracking transmissions. The framework presented in this study will significantly improve the performance of ultrasonic tracking, leading to faster image acquisition rates and increased localisation accuracy

Video-Rate All-Optical Ultrasound Imaging

All-optical ultrasound imaging, where ultrasound is generated and detected using light, has recently been demonstrated as a viable modality that is inherently insensitive to electromagnetic interference and exhibits wide bandwidths. High-quality 2D and 3D all-optical ultrasound images of tissues have previously been presented; however, to date, long acquisition times (ranging from minutes to hours) have hindered clinical application. Here, we present the first all-optical ultrasound imaging system capable of video-rate, real-time two-dimensional imaging of biological tissue. This was achieved using a spatially extended nano-composite optical ultrasound generator, a highly sensitive fibre-optic acoustic receiver, and eccentric illumination resulting in an acoustic source exhibiting optimal directivity. This source was scanned across a one-dimensional source aperture using a fast galvo mirror, thus enabling the dynamic synthesis of source arrays comprising spatially overlapping sources at non-uniform source separation distances. The resulting system achieved a sustained frame rate of 15 Hz, a dynamic range of 30 dB, a penetration depth of at least 6 mm, a resolution of 75 µm (axial) by 100 µm (lateral), and enabled the dynamics of a pulsating ex vivo carotid artery to be captured

Enhancement of instrumented ultrasonic tracking images using deep learning

PURPOSE: Instrumented ultrasonic tracking provides needle localisation during ultrasound-guided minimally invasive percutaneous procedures. Here, a post-processing framework based on a convolutional neural network (CNN) is proposed to improve the spatial resolution of ultrasonic tracking images. METHODS: The custom ultrasonic tracking system comprised a needle with an integrated fibre-optic ultrasound (US) transmitter and a clinical US probe for receiving those transmissions and for acquiring B-mode US images. For post-processing of tracking images reconstructed from the received fibre-optic US transmissions, a recently-developed framework based on ResNet architecture, trained with a purely synthetic dataset, was employed. A preliminary evaluation of this framework was performed with data acquired from needle insertions in the heart of a fetal sheep in vivo. The axial and lateral spatial resolution of the tracking images were used as performance metrics of the trained network. RESULTS: Application of the CNN yielded improvements in the spatial resolution of the tracking images. In three needle insertions, in which the tip depth ranged from 23.9 to 38.4 mm, the lateral resolution improved from 2.11 to 1.58 mm, and the axial resolution improved from 1.29 to 0.46 mm. CONCLUSION: The results provide strong indications of the potential of CNNs to improve the spatial resolution of ultrasonic tracking images and thereby to increase the accuracy of needle tip localisation. These improvements could have broad applicability and impact across multiple clinical fields, which could lead to improvements in procedural efficiency and reductions in risk of complications

Ultrasound 3D reconstruction of malignant masses in robotic-assisted partial nephrectomy using the PAF rail system: a comparison study

PURPOSE: In robotic-assisted partial nephrectomy (RAPN), the use of intraoperative ultrasound (IOUS) helps to localise and outline the tumours as well as the blood vessels within the kidney. The aim of this work is to evaluate the use of the pneumatically attachable flexible (PAF) rail system for US 3D reconstruction of malignant masses in RAPN. The PAF rail system is a novel device developed and previously presented by the authors to enable track-guided US scanning. METHODS: We present a comparison study between US 3D reconstruction of masses based on: the da Vinci Surgical System kinematics, single- and stereo-camera tracking of visual markers embedded on the probe. An US-realistic kidney phantom embedding a mass is used for testing. A new design for the US probe attachment to enhance the performance of the kinematic approach is presented. A feature extraction algorithm is proposed to detect the margins of the targeted mass in US images. RESULTS: To evaluate the performance of the investigated approaches the resulting 3D reconstructions have been compared to a CT scan of the phantom. The data collected indicates that single camera reconstruction outperformed the other approaches, reconstructing with a sub-millimetre accuracy the targeted mass. CONCLUSIONS: This work demonstrates that the PAF rail system provides a reliable platform to enable accurate US 3D reconstruction of masses in RAPN procedures. The proposed system has also the potential to be employed in other surgical procedures such as hepatectomy or laparoscopic liver resection

Soft optically-tuneable fluorescence phantoms based on gel wax and quantum dots: a tissue surrogate for fluorescence imaging validation

Fluorescence-guided brain tumour resection, notably using 5-aminolevulinic acid (ALA)-induced protoporphyrin IX (PpIX) for high-grade gliomas, has been demonstrated to provide better tissue differentiation, thereby improving patient outcomes when compared to white-light guidance. Novel fluorescence imaging devices aiming to increase detection specificity and sensitivity and targeting applications beyond high-grade gliomas are typically assessed by measurements using tissue-mimicking optical phantoms. The field currently lacks adequate phantoms with well-characterised tuneable optical properties. In this study, we developed soft tissue-mimicking fluorescence phantoms (TMFP) highly suitable for this purpose. We investigated: 1) the ability to independently tune optical and fluorescent properties; 2) the stability of the fluorescence signal over time; and 3) the potential of the proposed phantoms for imaging device validation. The TMFP is based on gel-wax which is an optically transparent mineral-oil based soft material. We embedded TiO2 as scattering material, carbon black oil-paint as background absorber, and CdTe Quantum Dots (QDs) as fluorophore because of its similar fluorescence spectrum to PpIX. Scattering and absorption properties were measured by a spectrophotometer, while the fluorescence was assessed by a wide-field fluorescence imaging system (WFFI) and a spectrometer. We demonstrated that: 1) the addition of QDs didn’t alter the phantom’s scattering which was only defined by the concentration of TiO2, whereas its absorption was defined by both QDs and colour oil paint; 2) the measured fluorescence intensity was linearlyproportional to the concentration of QDs; 3) the fluorescence intensity was stable over time (up to eight months); and 4) the fluorescence signal measured by the WFFI were strongly correlated to spectrometer measurements

Patient-Specific 3D Printed Models for Education, Research and Surgical Simulation

3D printing techniques are increasingly used in engineering science, allowing the use of computer aided design (CAD) to rapidly and inexpensively create prototypes and components. There is also growing interest in the application of these techniques in a clinical context for the creation of anatomically accurate 3D printed models from medical images for therapy planning, research, training and teaching applications. However, the techniques and tools available to create 3D models of anatomical structures typically require specialist knowledge in image processing and mesh manipulation to achieve. In this book chapter we describe the advantages of 3D printing for patient education, healthcare professional education, interventional planning and implant development. We also describe how to use medical image data to segment volumes of interest, refine and prepare for 3D printing. We will use a lung as an example. The information in this section will allow anyone to create own 3D printed models from medical image data. This knowledge will be of use to anyone with little or no previous experience in medical image processing who have identified a potential application for 3D printing in a medical context, or those with a more general interest in the techniques

Source density apodisation in 2D all-optical ultrasound imaging

In this work, an all-optical ultrasound imaging system that is capable of synthesising arbitrary source aperture geometries is presented. This capability is achieved by delivering focussed excitation light onto a spatially extended generating surface, where ultrasound is generated photoacoustically. Using a scanning mirror, the position of the resulting acoustical source was continuously varied to scan an aperture. This system exhibited sufficient sensitivity to acquire 2D images of clinically relevant tissue in under a second, as demonstrated on a tissue-mimicking phantom. The flexibility in the source array geometry was demonstrated through the implementation of two source array geometries on the same system, which allowed for the direct comparison of the image quality. It was shown that applying source density apodisation to obtain an aperiodic source array resulted in an improvement of up to 5 dB in image contrast, as compared to using a conventional, periodic array exhibiting the same number of sources and spatial extents